In civil engineering education, hand-drawn sketches of structural systems are commonly used for response analysis. Based on these sketches, students may determine the response of the structures either by hand calculations or via some specialized software through the construction of finite-element models. The former method is prone to errors, while the latter method could be time-consuming for inexperienced students. It would be convenient if the information within the hand-drawn sketches could be automatically converted into computer recognizable objects for further structural analysis. To address this issue, a novel method entitled Image-based Structural Analysis (ISA) is proposed as a proof of concept to determine the response of a linear-elastic structure directly from the image of a hand-drawn sketch. A selective search algorithm and a deep convolutional neural network are adopted to detect relevant objects in the images. Based on the bounding boxes and the classes of the objects, a finite element model is constructed for further structural response analysis. This study demonstrates the proposed ISA method via several hand-drawn beams under loadings. Results show that the proposed method, which consists of the combination of artificial intelligence and semantic rules, can produce, and analyze structural models from hand-drawn sketches.