The sense of agency has been defined as the sense that I am the author of my own actions. This sense, however, is usually not reflected upon but instead pre-reflectively experienced. Experimental approaches usually measure the sense of agency by judgments or verbal reports, despite evidence that the sense of agency is not sufficiently assessed on such a reflective level. Here we sought to identify non-verbal measures of the sense of agency, particularly testing the relevance of physiological activity such as skin conductance and heart rate. Manipulating the visual feedback to an executed movement, we investigated how well physiological activity and other movement parameters differed between real and false feedback (i.e., between actual agency and non-agency), and how they related to accuracy of agency judgments. Skin conductance and heart rate did not differ between agency and non-agency situations; neither did they inform agency judgments. In contrast, movement onsets - particularly, discrepancies between feedback and movement onsets - were related to agency judgments. Overall, our results indicate weak visceral-somatic associations with the sense of agency. Thus, physiological activity did not prove to be an empirical indicator for the feeling of agency.