It’s one of my favorite lines (phrase) in the song.I loved the interview with Rob Mounsey in Rick Henry's The Carpenters Online Interviews. In it he talked about this track and Karen's contribution to it:
"I was so happy that she wanted to do my song...The only problem was that it was really a man's song to a woman...The second line was "I was only watching the flower in your hair...". Not being a congenital lyricist I was stumped for a substitute line and it was Karen who came up with the "only trying to memorize you there", which I thought was okay and probably the best thing we would come up with".
I think Karen's lyric suggestion was more than okay! It fits perfectly with the preceding line "I really don't mean to stare".