We are familiar with the term iyun tefillah from our daily davening, where iyun tefillah is included amongst "the things whose fruits we eat in this world [but] whose principle [reward] awaits in the world to come". The term iyun implies to explore or investigate and thus, iyun tefillah would seem to indicate the seriousness of prayer, and the seriousness with which prayer is treated. Those who do the hard work and maintain the focus necessary to properly pray will be greatly rewarded. The difficulty of doing such is expressed in the teaching of Rav Amram in the name of Rav that there are "three sins that no person escapes from every day; sinful thoughts, iyun tefillah, and lashon hara" (Bava Batra 164b). To be fully focused on prayer three times a day is something almost beyond human reach--but reach we must.
Rav Yitzchak's teaching reflects the danger of beseeching G-d in prayer in an unfocused manner. Such is liable to bring our iniquities to G-d's mind.
Strictly speaking, prayer that lacks kavanah, proper focus, is no prayer at all; and at times, the halacha demands that such prayer be repeated with kavanah. The only reason we no longer do so is because we fear there will be little or no improvement the second time around (Rama Orach Chaim 101:1). Thus, better not to pray again (or maybe even the first time). The Gemara forbids one to pray if one's mind is focused on other matters. If we applied this principle rigorously, one might never pray; and thus, in practice we do pray--but only once[1].
Yet as logical as the above sounds, Rashi and Tosafot (who quotes this interpretation, s.v. v'iyun tefillah) reject this explanation. The Gemara does not say that a lack of iyun tefillah calls our sins to mind, but rather, that iyun tefillah itself is something to avoid. They interpret iyun tefillah to mean excessive confidence, and perhaps even excessive time spent in prayer; something (perhaps counter-intuitively) that can be most harmful.
It is the height of arrogance to think that, just because I pray long, hard, and with true devotion and fervour, that G-d need or should accede to my requests[2]. "One who extends his prayers and ismeayen bah, expects fulfillment, in the end will have a pained heart" (Berachot 55a) when all does not go according to plan. If anything, such prayer might do the opposite. G-d is just as likely to say, who is this person who spends so much time and prays so hard? And when G-d goes to investigate, the results may be worrisome. Not unlike an unprepared student who avoids anything that might bring the teacher's attention upon them, one is probably better off not to draw G-d's attention to oneself in prayer. The more we spend on iyun tefillah, the more G-d might be meayin, investigate us, often with poor results.
On this basis, Tosafot explains the link between iyun tefillah, a shaky wall, and calling for [Divine] judgment on one's fellow man, which also bring our sins to G-d's attention. All three are cases in which one is much too overconfident of having G-d's "protection", willing to take undue risks, and assuming his merit will allow his fellow to be punished because of him. A little humility can go a long way.
Tefillah reflects the complete loneliness and powerlessness of man[3]. We must approach G-d like an infant, who is totally dependent on his parents for his survival. One who has any degree of certainty and confidence in his prayers is taking undue risks and may be causing the exact opposite effect. We may--nay, must--pray that G-d answers our prayers, but we must assume that we are not yet worthy of such. And making ourselves more worthy is what Rosh Hashanah is all about.
[1] This is similar to the donning of tefillin, which requires an even higher degree of focus--something very few can achieve for any length of time, if at all. We are thus strongly discouraged from wearing tefillin beyond what is absolutely necessary.
[2] If and how this passage can be reconciled with the teaching of Rabbi Chanina ben Dosa, who would "pray for the sick and say, 'This one will live and this one will die...if the prayer is smooth on the lips, I know it will be accepted, and if not, I know that it will be ripped up'" (Brachot 34b) is something we will need to explore at another time.
[3] It is feeling that makes prayer so difficult. Modern man has done so much to enable us to control our fate that we, quite understandably, often feel little need for G-d. It is only when crisis hits that our perception of the great advances of society are shattered.