30% off your first 3 months

Get the deal

Definition

Computer use

Computer use is when an AI looks at a screen and operates it like a person would, by moving a cursor, clicking and typing.

computer useagentsscreen
  • The AI sees the screen, decides what to do, and acts with a cursor and keyboard.
  • It works in apps that were never built for it, because it uses them the way a person does.
  • For risky steps, like sending or paying, a careful system asks first.

Computer use means an AI operating a computer through its screen. Instead of calling a program's hidden functions, it looks at what is displayed, works out what each part is, and then moves a cursor, clicks, scrolls and types, the same way you would. Because it only needs the screen, it can work in almost any app.

Example: opening a page and clicking a button

Say you ask an AI to download this month's statement from your bank's website. It follows a loop:

  1. It looks at the screen and sees a browser with a home page.
  2. It decides the next move: open the statements page. It moves its cursor to the link and clicks.
  3. It looks again, finds the button labelled "Download", and clicks it.
  4. It checks that a file appeared, and reports back.

Each cycle of look, decide, act is one step. If the page asks you to sign in, a careful system hands that to you. DuckyHelper, for example, is built not to type your passwords or card numbers, and you can stop it or take control at any moment. Its agent work is in the Mac app.

Why it matters

Most software has no way for another program to control it, but all of it has a screen. Computer use opens that software to automation, at the cost of being slower than calling a function directly. Because the AI can act, it needs limits: asking before steps that cannot be undone, which is the idea of a human in the loop. Read what computer use means for AI assistants for more.

Further reading

Reviewed . Next review .

Owner: DuckyHelper

Questions and jobs

Do anything

Ask it a question or hand it a job, in any app.

Doing jobs for you is in the Mac app. Asking works on Mac, Windows and Linux.