How to get image data from embedded iframe?

Code Help HTML/CSS/JS
1

I’ve tried to use ImageData to get the color of a pixel in an embedded <iframe>. However, it only works with <canvas>. Is there a way to put <iframe> into <canvas>, or at least some other way of reading the ImageData of a website, regardless of content?

https://replit.com/@arn5891/running-python-in-html

2

tl;dr: No

This would be possible if you had access to the website you are embedding, and in this case, you don’t. (unless you are a Replit staff member and have write access to the code of replit.com lol)

The only way to communicate with an iframe is if:

  1. It’s hosted on the same origin as the parent website (in this case, the parent website’s origin is running-python-in-html.arn5891.repl.co and the iframe is replit.com)
  2. You have access to the code of the iframe. In this case, you could pass messages between the two windows using window.onmessage and window.postMessage

If you really wanted to, you could use getDisplayMedia to capture the content of the user’s screen and get pixel color from there

1 Like
3

getDisplayMedia prompts the user too much- is there a way to convert iframe to an image?

4

That’s pretty much the point. If the iframe is on a different origin, then it would be a serious security issue if the parent window could read it or even take a screenshot of it.

If you somehow do get access to the DOM of the iframe (like if it’s on the same origin) then you could use html2canvas

5

I know about the security issues with iframe. You also said it’s impossible to access DOM unless I’m the developer of Replit.

Is there a way to convert iframe to an html2canvas image?

6

Using js you could fetch whatever website then use html2canvas. Although it would still pose some CORS issues. You could also just make an backend api and remove all CORS issues.

7

html2canvas constructs an image by reading the dom. I assume there would be a big blank space where the iframe is.

You could use @dragonhunter1’s solution, but I don’t think it would work for your use case where the user interacts with the iframe

1 Like
8

All I need is real-time imagedata of whatever the user is doing in the iframe, I can hide the canvas in the background and still have the iframe up. Is there a general way of getting a screenshot of any element?

9

No, and for good reason - that would be a security issue. Imagine if a malicious website had an iframe to a bank website account page which had confidential information on it. The malicious website would be able to screenshot it and steal the user’s data.

AFAIK, the only ways of rendering a screenshot of the page are:

  1. html2canvas, which requires access to the DOM - which you don’t have for the iframe
  2. getDisplayMedia, which prompts the user
10

Yeah, every single thing I’ve looked up talks about the security restrictions. I don’t know jQuery, but a lot of sites recommend using it. I’ve already installed a jQuery module, is there anything else I need to set it up?

update- nvm, you already talked about jQuery with DOM. I’ll just use PyScript with the canvas instead of iframe, it removes a lot of problems.