Lord Swaminarayan first came to Amdavad as Nilkanth Varni in 1799 staying at Lake Kankaria. He visited the city many times, often celebrating the festival of Fuldol here. He frequently bathed in the River Sabarmati, passing en-route, through a fruit orchard in the area known as Shahibaug. Lord Swaminarayan built the first ever Swaminarayan mandir in the Kalupur area of Amdavad in the year 1822, installing the images of Nara and Narayan.
Lord Swaminarayan gave darshan to a devotee, Ashabhai, that a mandir of true upasana would soon spring forth in Amdavad. It was also Shastriji Maharaj's wish that an Akshar Purushottam mandir be built in Amdavad. To this end, an old bungalow was purchased in the area of Shahibaug - the fruit orchard which had been sanctified by Lord Swaminarayan. The bungalow was converted into a small Hari mandir and a small image of Lalji Maharaj was installed. At this site, Yogiji Maharaj later constructed a 5-spired mandir and fulfiled his guru's wish.