Or, put more clearly, if someone who is used only to a life of crime becomes interested in Jesus and steals a Bible so they can find out more about him, is that wrong? I remember hearing someone speak who did just that - it may have been Tal Brooke, but I'm not sure, so I'll call the person Jimmy. Once they became a Christian, they later returned the Bible.
Argument that it is wrong
The Bible quite clearly says "Do not steal". It's in the 10 Commandments, Jesus repeated it, Paul repeated it. Stealing is wrong.
Romans 2 is clear that the law applies only to those who know it, and those who don't are judged by what is seen of God's Law in their own consciences. Jimmy may honestly not have known that God didn't want him to steal the Bible. And yes, that probably means that Jimmy is guilty of stealing an awful lot of things previously, but in this case it's not actually his fault.
Furthermore, Paul says that "everything that does not come from faith is sin" (Rom 14:23). If we take that as a definition of sin, which it works well as, the stealing the Bible could actually be from a nascent faith and hence be the best thing Jimmy has yet done in his life.
The discussion was spawned by mention of this magnificent song, which was of course released for free download...