Am I being a baby about my job?

You need to learn the job otherwise you might not have one. The only people you need to worry about impressing is your boss. You will make friends with the co-workers if it is meant to be. Don't try to hard to get in the "in crowd". Everywhere has cliques and it seems like people never really grow up after high school.