Yes it may be a single allocating boot adresse probleme. Did you use an existing distribution, or did you try to compile yourself the code ? I have got this kind of probleme on the OMAP5432 when i tried myself to compile my own SD boot sequence.
Or unfortunately it may be a dysfunctioning component on the board ?