A new app that breaks language barriers

A new app that breaks language barriers

ROMA – NATIONAL University of Lesotho (NUL) students are developing a computer app that will translate sign language into spoken language. This is an ambitious project led by Lika Masoebe and Mokhutli Letsae, fourth year Computer Science students. Their project has captured the attention of the World Academy of Science, Engineering and Technology which has now invited the NUL scientists to present it at the 20th International Conference on Artificial Intelligence, Machine Learning and Soft Computing in Copenhagen, Denmark. “We are excited about this,” Letsae says.

“One of our supervisors will make an oral presentation concerning the strides we have already made at that esteemed conference.” The supervisors are Napo Mosola and Molete Sekese. It is not easy to be part of such an elite event. Of all the many presentations that will take place in that conference from all over the world, only three are from Africa – two from South Africa and one from Lesotho. These guys will effectively represent Africa there. There must be something to this project that has caught the attention of the world’s finest minds.

What is it? “It is a project in which we use computer systems to transform sign language into everyday languages,” Masoebe says. For instance, it can take American Sign Language (or any other sign language) into English or any other language. It is aimed at a global audience. Anyone who understands what that means already knows this is no easy task. In fact it is very, very difficult to achieve. “But,” Masoebe says, “we want to break the barrier between the vocally impaired and the able society.”

By the way, have you noticed that barrier? Maybe not. And that is precisely why these folks are concerned. According to them, most of us are just not worried about the vocally impaired, we take no notice. Many people won’t even bother to learn sign language so they can communicate with them, Masoebe says. But just like the rest of us, the vocally impaired want to communicate. They want to be listened to. They want to freely express themselves to a wider society. They are no different from us. Human interpreters are normally the ‘go-betweens’.

But they are not only rare, they can also be expensive. So how about an interpreter who will not only be always available, but also come at a low cost? This is where the app under development will fit in. This is how it works. For now, the computer scientists are focusing exclusively on transforming sign language letters into corresponding English alphabet letters (translating words will be even more of a monumental task). For instance, take an “A” in sign language into an “A” in English. That sounds simple but, hey, it is all but simple. So here we go. A vocally impaired person makes a sign for “A”. Then a camera picks up the sign and two things happen. The gesture is first detected, then, recognised.

What does that mean? When a certain sign is made with a hand gesture, the machine will first have to detect that gesture as being part of a sign language. There is always an unlimited number of gestures which are not necessarily sign language and should be ignored. Once the computer detects a gesture, it draws a square around that gesture. Well, that is a big deal. “We are going to Copenhagen precisely because we have been able to achieve this part,” Letsae says. Think about what it takes to achieve this. We have to enter a territory of the Artificial Intelligence or Machine Learning to understand this.

In Masoebe’s own words, “We are literally teaching the machine to do what we want it to do.” How is that even possible? Think about this uncomfortable reality the scientists unfortunately have to confront. No two people will make exactly the same gestures even if they sign the same letter. In fact, even a single person may sign the same letter differently each time. To complicate matters, no two hands are exactly the same, some are short, some are thin, some are black, some are white, well, you have a complete set of variety you can imagine. But just as your brain is designed to detect a gesture despite that variety, so should the machine.

“Instead of defining a single gesture, the computer should be able to define and hold a set of possibilities that refer to the same thing,” Masoebe says. “It has to be able to recognise certain features that are common in those numerous gestures, despite all the startling amount of differences.” That is why it has to be taught, just like a child. Then there is recognition. The scientists are not there yet, but they are getting extremely close.

Remember they have a paper on detection alone. In recognition, the computer is not only detecting a gesture, it is now saying, oh, yes, this gesture actually means “A” in your language. At the point, those walls that separate us from our loved ones, the vocally impaired, will start to crumble — for good.

Own Correspondent

Previous SADC extends standby force’s tenure
Next LNDC pushes for WRAP certification

About author

You might also like

Local News

Smally Trading won uniform case

MASERU – ON July 26, we published a story headlined, Uniform deal haunts police, in which we said businesswoman, Leonia Mosothoane’s firm, Smally Trading Company (Pty) Ltd, had lost a case

Local News

Thabane set for return

Staff Reporter MASERU EXILED opposition leaders will be home at the end of August, if SADC gets its wish. This is after the SADC Double Troika Summit held in Botswana

Local News

NUM rep picked for questioning

Staff Reporter MASERU – A REPRESENTATIVE of the South Africa-based National Union of Mineworkers (NUM) in Lesotho, Montoeli Masoetsa, was yesterday picked up for questioning by the police. Masoetsa was