The castle on the confluence of the Nemunas and the Neris rivers was built in the 14th century to fend off the attacks of Crusaders. It is one of the first stone castles in Lithuania, and also the only one with two rows of defensive walls. Around this Gothic defensive structure, a settlement was built that grew into the current city.  The castle is mentioned in written sources for the first time in 1361.