When this song came out in 1966, there was a widespread rumor that it was about getting high on banana skins. The idea was that you scraped the fibers off of a banana skin and cooked them over a low fire. This was supposed to release the hallucinogenic qualities. Of course, it was never true.
In an interview with the June 18, 2011 edition of the NME, Donovan was asked what the song was actually about. He replied: "Quite a few things. Being mellow, laid-back, chilled out. 'They call me Mellow Yellow, I'm the guy who can calm you down.' [John] Lennon and I used to look in the back of newspapers and pull out funny things and they'd end up in songs. So it's about being cool, laid-back, and also the electrical bananas that were appearing on the scene - which were ladies vibrators."
Paul McCartney can be heard as one of the background revellers on this track, but the "quite rightly" whispering answering lines in the chorus is not McCartney but rather Donovan himself.
There doesn't seem to be anything here yet