Hello world!

Welcome to WordPress.com. This is your first post. Edit or delete it and start blogging!

Posted in Uncategorized | 1 Comment

Call for Registration: ACM Multimedia 2010 Tutorials

ACM Multimedia 2010 Tutorial Program – Call For Registration 

ACM Multimedia 2010 tutorial program will provide in-depth learning and discussion opportunities on state-of-the-art multimedia research and development.  All tutorials will be held on the first day (Oct. 25, 2010) of the main conference. This year we are offering 14 tutorials that cover current and emerging topics from the four major tracks of the conference. The detailed information of all the tutorials can be found at: http://www.acmmm10.org/program/tutorials/. You can register for any of the tutorials through the main conference registration site at http://www.acmmm10.org/attendees/registration.

Posted in Research | Leave a comment

Interactive Image Search: Search by Color Sketch – Just Released in Microsoft Bing Image Search

On June 3, Microsoft released a new version of its Bing search engine. Among the new functionalities, an interactive image search interface provides a totally new capacity of searching images on the Web. This article will introduce this feature with a couple of interesting examples.

The new feature is based on a research effort from Microsoft Research Asia (MSRA). Last year, the two major contributors for this technique from MSRA, Xian-Sheng Hua and Jingdong Wang, has reported this technology using a number of screenshots of a prototype in MSRA Blog (Chinese version). Now this interesting technique is in Bing image search that everyone can try.

Originally the feature is called “search by color map” in the prototype. But now it is called “Specific color” and “Filter using sketch” in Bing image search. We can call it “image search by color sketch”.

In this article, we will first introduce the steps of using this feature, which has three functionalities, and then present more details and tips of using this feature. And last we will talk a little bit about the technology behind.

Table of Content

How to Use Search by Color Sketch

More Examples

Search people (“Lady Gaga”) 
Search car (“BMW”)

About the Underlying Technologies


How to Use Search by Color Sketch

1. Visit http://www.bing.com and click “Images”, or directly visit: http://www.bing.com/images or http://image.bing.com;

2. Check the top-right corner of the web page and change the region of the service to “United States” if currently it is not. Currently this feature is only enabled in US region, but can be used worldwide as long as you changed the region to “United States”;

3. Type a query in the search box and press search button. Now we use “apple” as an example.

4. Click “COLOR” on the left panel (if it is not expanded), then you will see a menu “Specific color”. This is the entrance of the new feature. Currently it is hidden and hope it will be placed somewhere easier to be found (Figure 1).

(a) The whole page

(b) Enlarged: the bottom-right corner of the page

Figure 1: Open the entrance of the new feature.

5. After clicking, you get a panel at bottom-left part of the page (Figure 1), where you can try the first functionality: search by dominant color. That means, you can get “flower” related images but the dominant color of the images would be the one you clicked. For example, if you click the small blue block, you will get results like Figure 2 (mostly blue flowers or flowers’ color close to blue) (Figure 2)

Figure 2: Blue flowers (“zoom small” was clicked so much more images can be shown in one page, which can be set on top-right of the image area)

6. The second and third functionality needs open the drawing panel by checking “Filter using sketch” in the color panel. Then you can choose colors and draw the color sketch of the desired image (Figure 3). Please note this is not for drawing contour of a cat, dog, house or other objects, but only the color “map” of desired images. For example, we may want to find purple flowers with green leaf – that’s what Figure 3 is showing. After drawing the color sketch, press “Apply” to get new search results. We can call this “search by drawing” or “search by color sketch”.

Figure 3: Purple flowers with green leaves.

7. The third functionality could be called “search by modified example”. That means, instead of drawing on the empty panel, we can drag-n-drop an image from the search results to the panel. We can directly click “Apply” to search similar images (Figure 4), or we can draw something to change the example before search. For example, we use the eraser to ease the green leaves and then draw some yellow blocks, which indicates that I’d like to see some yellow followers or parts besides the purple ones (Figure 5).

Figure 4: Search by an example (drag-n-drop an image from the search results).

Figure 5: Search by modified example (purple flowers but with something yellow). The first image in the results will remain the same as the originally selected image no matter how you modified it. This is by design, so people will always know which image he or she has chosen.

There are hyperlinks embedded in the example pictures above (and also those in next section). You can click it to see what it looks like in Bing image search. However, the results may be slightly different (or even much different if you read this article long after June 2010) with the pictures showing here as Bing’s image index is keeping updating. If the links are invalid, you may try it manually using the same term and similar drawing, or any other drawings you like.

Below figure shows the functions of the button in the panel (Figure 6).

Figure 6: Functions of the buttons.

More Examples

We will show you a number of examples, based on which you will learn how to well express you intent through drawing color sketch.


Type “grass” in Bing image search you will get something like Figure 7.

Figure 7: Original results of “grass” in Bing image search.

If want more sky on the top and less grass at the bottom, you can draw something to express this intent and get something like Figure 8.

Figure 8: “Grass” images with more sky on top and less grass at bottom.

Or you want less sky on top and more grass at bottom (Figure 9):

Figure 9: “Grass” images with little sky and lots of grass.

Sometimes you may want to find “orange grass” (Figure 10), or you find one of the orange grass looks more like what you want (drag it to the panel as an example) (figure 11).

Figure 10: Orange grass.

Figure 11: Search by example “grass”

Sometimes, you may want to get some grass images with clean background, so it will be easier to use it. This can be achieved by set the background to white or black or any other possible background color (Figure 12).

Figure 12: Grass images with clean/white background

Another example of searching by example: grass with orange sky (Figure 13), and modified example (similar grass with blue sky) (Figure 14).

Figure 13: Search by example (grass with orange sky area).

Figure 14: Search by modified example (similar grass but with blue sky). Again, the first image in the results will remain the same as the originally selected image no matter how you modified it. This is by design, so people will always know which image he or she has chosen.


Original results of searching “flower” in Bing image search (Figure 15). Blue flowers or some flower related examples have been shown in Figure 2, 3, 4 and 5. Here are more examples: purple flower (Figure 16), yellow flower (Figure 17), yellow flower with dark background (Figure 18), search by example (an yellow flower with dark background) (Figure 19), and search by modified example (yellow flower with green background)(Figure 20).

Figure 15: Original results of “flower” in Bing image search.

Figure 16: Purple flowers (by selecting the dominant color).

Figure 17: Yellow flowers (indicated by yellow blocks in the center area).

Figure 18: Search by drawing (yellow flowers with dark background, indicated by yellow blocks at center area and black blocks at the border).

Figure 19: Search by example (yellow flower with dark background).

Figure 20: Search by modified example (change the background to green, indicating leaves or grass).


When you saw a flag of a country when watching world cup, but you don’t know what the country of the flag, you may use Bing image search to find it out. Below shows several results (Figure 21 to Figure 27) – after drawing the flag roughly, you get the real flag images. Mouse-over or click the desired result, you will know the country of the flag.

Figure 21: China flag.

Figure 22: India flag (and other similar flags, such as Miami city flag and Niger flag)

Figure 23: Niger flag (and other similar ones).

Figure 24: Ghana flag.

Figure 25: Mauritius flag (search by example).

Figure 26: Gambia flag (search by modified example).

Figure 27: Danmark flag and similar flags (search by drawing).

My son (2.5 years old) is able to recognize more than 80 flags of different countries, but I can only recognize 10+. Now with Bing’s “search by color sketch”, I can compete with him now, thought the speed is much slower than him 🙂

Search people (“Lady Gaga”)

“Lady Gaga” is very popular recently. Figure 28 shows the initial results in Bing. Figure 29, 30, 31, and 32 shows the images of Lady Gata walking on red carpet and wearing in different colors.

Figure 28: Original results of “Lady Gaga” from Bing image search.

Figure 29: Lady Gaga walking on red carpet (indicated by a red line at the bottom).

Figure 30: Lady Gaga walking on red carpet and wearing in black (indicated by a red line at the bottom and some black blocks near the center.

Figure 31: The same as above but wearing in white (sometimes you need to adjust the size and/or position of the color blocks to get better results).

Figure 32: Walking on red carpet while background is white.

Figure 33: Search by an example (wearing in black and blue).

Figure 34: Dark background.

Figure 35: Dark background with larger face area.

Figure 36: Search by another example with dark background.

Search car (“BMW”)

Figure 37 to Figure 43 show a couple examples for search color of different colors and different background.

Figure 37: Original results of “BMW” of Bing.

Figure 38: Blue BMW.

Figure 39: Red BMW.

Figure 40: Red BMW on lawn.

Figure 41: Red BWM on lawn with blue sky.

Figure 42: Search by modified example. Originally the car is black (i.e., the first image in the result list). Now it was changed to red – we got red cars in similar background.

Figure 43: The same as the above example, but we change the color of the car to orange.

About the Underlying Technologies

This is not re-ranking, not conventional “query by sketch”, not just user-interface. It should be the first web-scale “image search by drawing” system. When drawing is provided, a new query based on the text query and drawing will be issued to search new images from the entire Bing index.


(To be continued – some high-level introduction of the technology and the differences compared with existing work, as well as some tips of using the feature.)

By Xian-Sheng Hua, Beijing, June 2010

Posted in Research | 8 Comments

Microsoft Academic Search (paper search)


 Try it. Still in beta.

 (I was asked to give a try by some people working on it :))


Good UI. Support author search/rank list, conference search/rank list, journal search/rank list, and some other cool features (try it J)

For example, top ranked authors in multimedia area:

http://academic.research.microsoft.com/CSDirectory/Author_category_13.htm  (hope it was done automatically)

And it also support author profile page, for example: http://academic.research.microsoft.com/Author/951346.aspx 

However, the big problem is it cannot differentiate different people with the same name either. But seems there is no a search engine can solve this problem currently.



Dataset is not complete enough. Missing papers, citations, and indexes (such as g-index and h-index) are much blow the correct ones.

For the nice feature mentioned above (author profile), the downside is that it also provides a photo of the author, however, the listed papers may come from different people with the same name. The person whose papers are listed but the photo is not his/her may get angry about it 🙂


I believe they are improving, very quickly.I heard they are updating data every week and updating features every two months.

Posted in Research | 1 Comment



















  (摘自香港《大公报》 作者:志宏)


与《菜根谭》上的这句可互相印证:“甘辛膿肥非真味,真味只是淡,神奇卓異非至人,至人只是常。”不要忘了自己最初的目标,不要迷失在为达到目标而选择路径上 – 那还可能不是最好的路径。

《老子》第四十一章原文:“上士闻道,勤而行之;中士闻道,若存若亡;下士闻道,大笑之。不笑不足以为道。故建言有之: 明道若昧;进道若退;夷道若纇;上德若谷;大白若辱;广德若不足;建德若偷;质真若渝;大方无隅;大器晚成;大音希声;大象无形;道隐无名。夫唯道,善贷且成。”

Posted in Uncategorized | 3 Comments


Posted in Research | 4 Comments


早起上班路上听广播,收音机里讲了刚刚辞世的季羡林老先生关于“国学大师”和“国宝”的几段文字,感触颇深 — 做学问当效季老:













媒体赞季老:虚怀若谷 对虚荣社会无声抗议 (原文





  记 者:现在社会上和学术界对您有些微词。
  记 者:您为什么要摘掉自己头上的光环?像是“国宝”、“泰斗”、“大师”这些称号?
  记 者:您觉得谁够格?
  记 者:您对这个民族做出的贡献,是有目共睹的。

Posted in Research | Leave a comment